Add persistence package #107
Conversation
jraddaoui
force-pushed
the
dev/issue-106-duplicates
branch
from
875ee0b to
41b4a87
Compare
Create a persistence package for database integration using Ent, with support for SQLite and MySQL databases. Define database schema with a single `sip` table with `name` and `checksum` columns that will be used to check for duplicates at the bigginging of the preprocessing workflow. The persistence layer will only be used when `checkDuplicates` is set to `true` in the configuration. Update the dev env/overlay to use an SQLite database and the Enduro env/overlay to use MySQL. Enable `cgo` at build time and install `gcc` and `musl-dev` in the build Docker image to be able to build an static binary that uses `github.com/mattn/go-sqlite3`. [skip-codecov]
jraddaoui
force-pushed
the
dev/issue-106-duplicates
branch
from
41b4a87 to
a1e1c51
Compare
jraddaoui
changed the title
|
This is ready for code review @djjuhasz. @mcantelon this could be a good PR to know a bit more about the persistence layer/package in other projects where we use Ent/Atlas. @sevein I'd also like that you take a look when you have some time, specially about the |
There was a problem hiding this comment.
Looks good to me @jraddaoui. 👍
I guess adding a database was inevitable, but it was nice to avoid it as long as we did. 😉
| @@ -9,6 +9,12 @@ stringData: | |||
| verbosity = 2 | |||
|
|
|||
| sharedPath = "/home/preprocessing/shared" | |||
| checkDuplicates = true | |||
There was a problem hiding this comment.
Don't we want checkDuplicates to be false in the dev environment?
| // be required, and a SIP that has already been processed will fail the | ||
| // preprocessing workflow. | ||
| CheckDuplicates bool | ||
| Persistence persistence.Config |
There was a problem hiding this comment.
I think there should be a blank line after CheckDuplicates and no blank line after Persistence.
| case "mysql": | ||
| db, err = OpenMySQL(ds) | ||
| case "sqlite3": | ||
| db, err = sql.Open(driver, ds) |
There was a problem hiding this comment.
Why not put all this in an OpenMySQL(ds) function like with MySQL?
| } | ||
|
|
||
| // OpenMySQL opens the MySQL database driver. | ||
| func OpenMySQL(ds string) (*sql.DB, error) { |
There was a problem hiding this comment.
Does this need to be exported?
This is the first of at least three PRs to address #106. The next PR will
do the actual check in workflow (hopefully it can happen before this
week release), and the other one will deal with versioned migrations.
Create a persistence package for database integration using Ent, with
support for SQLite and MySQL databases. Define database schema with
a single
siptable withnameandchecksumcolumns that will beused to check for duplicates at the bigginging of the preprocessing
workflow. The persistence layer will only be used when
checkDuplicatesis set to
truein the configuration.Update the dev env/overlay to use an SQLite database and the Enduro
env/overlay to use MySQL. Enable
cgoat build time and installgccand
musl-devin the build Docker image to be able to build an staticbinary that uses
github.com/mattn/go-sqlite3.